This paper introduces Deep4D a compact generative representation of shape and appearancefrom captured 4D volumetric video sequences of people. 4D volumetric video achieves highlyrealistic reproduction, replay and free-viewpoint rendering of actor performance from multipleview video acquisition systems. A deep generative network is trained on 4D video sequencesof an actor performing multiple motions to learn a generative model of the dynamic shapeand appearance. We demonstrate the proposed generative model can provide a compactencoded representation capable of high-quality synthesis of 4D volumetric video with two ordersof magnitude compression. A variational encoder-decoder network is employed to learn anencoded latent space that maps from 3D skeletal pose to 4D shape and appearance. Thisenables high-quality 4D volumetric video synthesis to be driven by skeletal motion, includingskeletal motion capture data. This encoded latent space supports the representation of multiplesequences with dynamic interpolation to transition between motions. Therefore we introduceDeep4D motion graphs, a direct application of the proposed generative representation. Deep4Dmotion graphs allow real-tiome interactive character animation whilst preserving the plausiblerealism of movement and appearance from the captured volumetric video. Deep4D motion graphsimplicitly combine multiple captured motions from a unified representation for character animationfrom volumetric video, allowing novel charact...